Appearance determination device and appearance determination method

ABSTRACT

An appearance determination device includes at least one processor. The at least one processor carries out a determination step of determining whether an object of determination is conforming or nonconforming, in accordance with a color image and a shape image, the color image being generated by photometric stereo and representing a color optical image of the object of determination, the shape image being generated by the photometric stereo and representing a shape of the object of determination.

This Nonprovisional application claims priority under 35 U.S.C. § 119 onPatent Application No. 2022-107804 filed in Japan on Jul. 4, 2022, theentire contents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present invention relates to an appearance determination device andan appearance determination method.

BACKGROUND ART

Photometric stereo is a technique for analyzing the appearance of anarticle with use of a plurality of light sources of differentorientations (see, for example, Non-Patent Literatures 1 and 2).

CITATION LIST Non-Patent Literature Non-patent Literature 1

-   R. J. Woodham, Photometric Stereo: A Reflectance Map Technique for    Determining Surface Orientation from Image Intensity. Proc. SPIE,    vol. 155, pp. 136-143, 1978.

Non-patent Literature 2

-   H. Hayakawa. Photometric stereo under a light source with arbitrary    motion. Journal of the Optical Society of America A, 11(11), 1994.

SUMMARY OF INVENTION Technical Problem

Photometric stereo is typically used for monochrome grayscale images,and is not intended to be applied to color images. Therefore, in typicalphotometric stereo, information on color (dependence of reflection oflight from the article on the wavelength of the light), which isimportant in determining whether the appearance of the article isconforming or nonconforming, is discarded.

An object of the present invention is to provide an appearancedetermination device and an appearance determination method that usecolor photometric stereo to determine whether the appearance of anobject of determination is conforming or nonconforming.

Solution to Problem

In order for the above object to be achieved, an appearancedetermination device in accordance with an aspect of the presentinvention includes at least one processor, the at least one processorcarrying out a determination step of determining whether an object ofdetermination is conforming or nonconforming, in accordance with atleast one selected from the group consisting of a color image and ashape image, the color image being generated by photometric stereo andrepresenting a color optical image of the object of determination, theshape image being generated by the photometric stereo and representing ashape of the object of determination.

Advantageous Effects of Invention

With an aspect of the present invention, it is possible to provide anappearance determination device and an appearance determination methodthat use color photometric stereo to determine whether the appearance ofan object of determination is conforming or nonconforming.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view of a configuration of an appearance determinationsystem in accordance with an embodiment of the present invention.

FIG. 2 is a flowchart of an example of an appearance determinationmethod in accordance with an embodiment of the present invention.

FIG. 3 is a flowchart of an example of a method for generating a colorreflection image (color image) and a normal image (shape image).

FIG. 4 is a flowchart of another example of the method for generating acolor reflection image (color image) and a normal image (shape image).

FIG. 5 is a flowchart of an example of a method for estimating a lightsource vector.

FIG. 6 is a view of an example of the approach to removing an abnormallight source vector.

FIG. 7 is a schematic view of a model.

FIG. 8 is a view illustrating examples of appearance determinationresults.

FIG. 9 is a view illustrating another example of appearancedetermination results.

DESCRIPTION OF EMBODIMENTS

The following description will discuss an embodiment of the presentinvention in detail. FIG. 1 is a view of a configuration of anappearance determination system 1 in accordance with an embodiment ofthe present invention. The appearance determination system 1 is a systemfor determining whether an object OB of appearance determination is aconforming item (a non-defective item), and includes an image-takingsection MP and an appearance determination device 10.

The image-taking section MP is an image-taking device for photometricstereo, and includes a darkroom BX, a plurality of light sources LS(1)to LS(n), and a camera CA. Hereinafter, the light sources LS(1) to LS(n)are collectively referred to as light sources LS(i). For photometricstereo, the image-taking section MP uses the light sources LS(i) ofdifferent orientations to take an image of the object OB ofdetermination, and generates a plurality of images IM of the object OBof determination.

The darkroom BX defines a space for placing the object OB ofdetermination and taking an image of the same. The darkroom BX has wallsfor cutting off outside light. In the darkroom BX, the object OB ofdetermination, the light sources LS(i), and the camera CA are placed.

The light sources LS(i) are lighting equipment used when the camera CAtakes an image of the object OB of determination. The light sourcesLS(i) are placed in positions and orientations such that the lightsources LS(i) illuminate the object OB of determination from respectivedirections different from each other. In order for color photometricstereo to be provided, the light sources LS(i) apply light containinglight of a first wavelength band, light of a second wavelength band, andlight of a third wavelength band that are different from each other.

Examples of the first wavelength band, the second wavelength band, andthe third wavelength band can include the wavelength bands of R (red), G(green), and B (blue). In the following description, the firstwavelength band, the second wavelength band, and the third wavelengthband are referred to respectively as R, G, and B, for ease ofunderstanding.

The camera CA is an image-taking device for taking an image of theobject OB of determination. Although a single camera CA alone isillustrated here for ease of understanding, a plurality of cameras CAmay be placed in positions and orientations so as to take images of theobject OB of determination from respective directions different fromeach other.

The appearance determination device 10 is, for example, a personalcomputer, and determines whether the object OB of determination is aconforming item (a non-defective item), in accordance with a pluralityof images IM of the object OB of determination taken by the image-takingsection MP.

The appearance determination device 10 includes a processor 11, aprimary memory 12, a secondary memory 13, an input-output interface (IF)14, a communication IF 15, and a bus 16. The processor 11, the primarymemory 12, the secondary memory 13, the input-output IF 14 and thecommunication IF 15 are connected to each other via the bus 16.

The secondary memory 13 has stored (stored in a nonvolatile manner)therein an appearance determination program P1 and a model M1. Theprocessor 11 loads, into the primary memory 12, the appearancedetermination program P1 and the model M1 stored in the secondary memory13. The processor 11 then carries out an appearance determination methodaccording to the instructions contained in the appearance determinationprogram P1 loaded in the primary memory 12. The model M1 loaded in theprimary memory 12 is used when the processor 11 carries out theappearance determination method.

Examples of a device that can be used as the processor 11 can include acentral processing unit (CPU), a graphic processing unit (GPU), adigital signal processor (DSP), a micro processing unit (MPU), afloating point number processing unit (FPU), a physics processing unit(PPU), a microcontroller, and a combination thereof. The processor 11can be referred to as a “computing device”.

Examples of a device that can be used as the primary memory 12 caninclude a semiconductor random access memory (RAM). The primary memory12 can be referred to as “main storage”. Further, examples of a devicethat can be used as the secondary memory 13 can include a flash memory,a hard disk drive (HHD), a solid state drive (SSD), an optical diskdrive (ODD), a floppy (registered trademark) disk drive (FDD), and acombination thereof. The secondary memory 13 can referred to as“auxiliary storage”. The secondary memory 13 may be incorporated in theappearance determination device 10, or may be incorporated in anothercomputer (e.g., a computer of a cloud server) connected to theappearance determination device 10 via the input-output IF 14 or thecommunication IF 15. Although provided by two memories (the primarymemory 12 and the secondary memory 13) in the present embodiment, thestorage in the appearance determination device 10 is not limitedthereto. The storage in the appearance determination device 10 may beprovided by a single memory. In this case, for example, a certainstorage area of the single memory may be used as the primary memory 12and another storage area of the single memory may be used as thesecondary memory 13.

To the input-output IF 14, an input device and/or an output deviceis/are connected. Examples of the input-output IF 14 include a universalserial bus (USB) interface, an advanced technology attachment (ATA)interface, a small computer system interface (SCSI) interface, and aperipheral component interconnect (PCI) interface.

Examples of the input device connected to the input-output IF 14 includethe camera CA. Data retrieved from the camera CA in the appearancedetermination method is inputted to the appearance determination device10 and stored in the primary memory 12. Further, examples of anotherinput device connected to the input-output IF 14 include a keyboard amouse, a touchpad, a microphone, and a combination thereof. Examples ofthe output device connected to the input-output IF 14 include a display,a projector, a printer, a speaker, a headphone, and a combinationthereof. Information to be provided to a user in the appearancedetermination method is outputted from the appearance determinationdevice 10 via the output device listed above. Like a laptop computer,the appearance determination device 10 may incorporate both a keyboardthat functions as the input device and a display that functions as theoutput device. Alternatively, the appearance determination device 10 mayincorporate a touch panel that functions as both the input device andthe output device, like a tablet computer.

To the communication IF 15, another computer is connected via a network,either through wire or wirelessly. Examples of the communication IF 15include an Ethernet (registered trademark) interface and a Wi-Fi(registered trademark) interface. Examples of a usable network include apersonal area network (PAN), a local area network (LAN), a campus areanetwork (CAN), a metropolitan area network (MAN), a wide area network(WAN), a global area network (GAN), and an internetwork containing acombination thereof. The internetwork may be an intranet, may be anextranet, or may be the Internet. Data retrieved by the appearancedetermination device 10 from another computer in the appearancedetermination method and data provided by the appearance determinationdevice 10 to another computer in the appearance determination method aretransmitted and received via the network listed above. The camera CA andthe appearance determination device 10 may be connected together via theinput-output IF 14, or may be connected via the communication IF 15.

Although a single processor (processor 11) is used to carry out theappearance determination method in the present embodiment, the presentinvention is not limited to this. The appearance determination methodmay be carried out with the use of a plurality of processors. In thiscase, the plurality of processors that cooperatively carry out theappearance determination method may be provided in a single computer andmay communicate with each other via the bus. Alternatively, theplurality of processors may be distributed among a plurality ofcomputers and communicate with each other via a network. For example, aprocessor incorporated in a computer of a cloud server and a processorincorporated in a computer owned by a user of the cloud server cancooperate to carry out the appearance determination method.

Although, in the present embodiment, the model M1 is stored in thememory (secondary memory 13) incorporated in the same computer as thecomputer in which the processor (processor 11) that carries out theappearance determination method is incorporated, the present inventionis not limited thereto. The model M1 may be stored in a memoryincorporated in a computer that is not the computer in which theprocessor that carries out the appearance determination method isincorporated. In this case, the computer incorporating the memory forstoring the model M1 can communicate via the network with the computerincorporating the processor for carrying out the appearancedetermination method. For example, the model M1 is stored in a memoryincorporated in a computer of a cloud server, and a processorincorporated in a computer owned by a user of the cloud server can carryout the appearance determination method.

Although the model M1 is stored in a single memory (the secondary memory13) in the present embodiment, the present invention is not limited tothis. The model M1 may be distributed among a plurality of memories soas to be stored in the plurality of memories. In this case, a pluralityof memories for storing the model M1 may be provided on a singlecomputer (which may or may not be a computer incorporating a processorthat carries out an appearance determination method), or may bedistributed among a plurality of computers (which may or may not includea computer incorporating a processor that carries out an appearancedetermination method). For example, the model M1 can be distributedamong a plurality of computers of a cloud server so as to be stored inthe memories incorporated in the plurality of respective computers.

The appearance determination device 10 includes at least one processor11, and carries out the appearance determination method. FIG. 2 is aflowchart of an example of the appearance determination method carriedout by the processor 11. The appearance determination method includes apreparation step (step S11), a generation step (step S12), and adetermination step (step S13).

The preparation step is a step carried out as required. For example, thepreparation step is a step of forming or strengthening the model M 1.Details of the preparation step will be described later. The generationstep is the step of generating a color image and a shape image of theobject OB of determination, in accordance with the plurality of imagesIM of the object OB of determination taken by the image-taking sectionMP. The plurality of images IM are images of the object OB ofdetermination irradiated with light from the plurality of light sourcesLS(i) of respective orientations different from each other. Thedetermination step is a step of determining whether the object OB ofdetermination is conforming or nonconforming, in accordance with atleast one selected from the group consisting of the color image and theshape image.

The object OB of determination is an article that is subjected toappearance determination. The object OB of determination is, forexample, a casting having been cast or a mold used in casting. Theobject OB of determination is not limited to a casting or a mold, andmay be another article.

The color image is an image generated by photometric stereo andrepresenting a color optical image of the object OB of determination.The color image is, for example, a color reflection image representingthe distribution of reflectances on the surface of the object OB ofdetermination, the color reflection image corresponding to the firstwavelength band (R), the second wavelength band (G), and the thirdwavelength band (B) of light that are different from each other.

In typical photometric stereo, an albedo image representing thedistribution of reflectances on the object OB of determination isgenerated in accordance with a monochrome grayscale image. Therefore, analbedo image is a monochrome grayscale image. In contrast, according tothe present embodiment, color photometric stereo is provided so that acolor reflection image is generated, the color reflection imagerepresenting the distribution of reflectances on the object OB ofdetermination in color. Details of the generation of the colorreflection image will be described later.

The shape image is an image generated by photometric stereo andrepresenting the shape of the object OB of determination. The shapeimage is, for example, a normal image representing the distribution ofnormal directions on the surface of the object of determination. Inaccordance with the plurality of images IM of the object OB ofdetermination taken by the image-taking section MP, a normal image isgenerated together with the color reflection image.

A conforming item is an object (e.g., a casting) that does not contain adefect. Examples of the defect include chipping, cracking, a dent, aprotuberance, and the presence of foreign matter (e.g., screw, paperscrap, rust, and sand). Conforming items are allowed to be non-identicalto each other due to asperities, color unevenness, the presence orabsence of burrs, etc. of the surface of the casting.

Here is a description of the generation step (step S12). FIG. 3 is aflowchart of an example of a method for generating a color reflectionimage (color image) and a normal image (shape image). The generationstep illustrated in FIG. 3 can roughly be divided into (A) an estimationstep (step S21) of estimating a light source vector L(i), (B) anacquisition step (step S22) of acquiring a plurality of images IM(i) ofthe object OB of determination, and (C) a generation step (step S23, S24a to S24 c, S25, and S26) of generating a color image and a shape imageof the object OB of determination. The details are as follows.

(A) Estimation Step of Estimating Light Source Vector L(i) (Step S21)

The processor 11 estimates a light source vector L(i) (the direction ofeach of the plurality of light sources LS(i) (step S21).

The light source vector L(i) is a vector representing the orientation ofthe light source LS(i) with respect to the object OB of determinationand the distance of the light source LS(i) to the object OB ofdetermination. The light source vector L(i) is used to create a colorreflection image (color image) and a normal image (shape image). Thisestimation means determining the orientation of each of the plurality oflight sources LS(i).

In photometric stereo, by measuring the positional relationship betweenthe light source LS(i) and the object OB of determination, the lightsource vector L(i) can be determined. However, this measurement isdifficult in some cases. In such cases, it is necessary to estimate thelight source vector L(i). The details of this estimation will bedescribed later.

(B) Acquisition Step of Acquiring Plurality of Images IM(i) of Object OBof Determination (Taking Images, Switching of Lighting) (Step S22)

The processor 11 acquires a plurality of images IM(i) of the object OBof determination (step S22). Prior to this acquisition, the object OB ofdetermination is placed in the darkroom BX.

By taking images of the object OB of determination via the camera CAwhile switching the light sources LS(i), a plurality of images IM(i) ofthe object OB of determination can be acquired. The plurality of imagesIM(i) are a plurality of images of the object OB of determinationirradiated with light emitted from each of the plurality of lightsources LS(i) having respective orientations different from each other.The images IM(i) are taken with use of light containing light of R(first wavelength band), G (second wavelength band), and B (thirdwavelength band) and are color images containing pixels of R, G, and B.

(C) Generation Step of Generating Color Image and Shape Image of ObjectOB of Determination (Steps S23, S24 a to S24 c, S25, S26)

In this generation step, a color image and a shape image of the objectOB of determination are generated in accordance with the plurality ofimages IM(i) and the determined directions (light source vectors L(i)).The generation step can be divided into the following steps: (1) a step(step S23) of extracting a plurality of R images, a plurality of Gimages, and a plurality of B images; (2) a step (steps S24 a to 24 c) ofgenerating an R reflection image, a G reflection image, and a Breflection image, and an R normal image, a G normal image, and a Bnormal image; (3) a step (step S25) of generating a normal image (shapeimage); and (4) a step of generating a color reflection image (colorimage) (step S26). The details are as follows.

(1) Extract Plurality of R Images IMr(I), Plurality of G Images Img(I),and Plurality of B Images IMb(I) from Plurality of Images Im(I) (StepS23)

The processor 11 extracts, from the plurality of images IM(i) (stepS23), a plurality of first images (a plurality of R images IMr(i))corresponding to the first wavelength band (R), a plurality of secondimages (a plurality of G images IMg(i)) corresponding to the secondwavelength band (G), and a plurality of third images (a plurality of Bimages IMb(i)) corresponding to the third wavelength band (B).

By extracting the pixels of R, G, and B from the color image IM(i), an Rimage IMr(i) constituted by the pixels of R, a G image IMg(i)constituted by the pixels of G, and a B image IMb(i) constituted by thepixels of B can be generated.

In typical photometric stereo, color is not considered and an albedoimage (the reflection image herein) is generated from a monochromegrayscale image, as described above. According to the presentembodiment, in order for color photometric stereo to be provided, the Rimages IMr(i), the G images IMg(i), and the B images IMb(i) areextracted from the images IM(i) to be individually processed. This makesit possible to provide color photometric stereo.

(2) Generation of R, G, and B Reflection Images and R, G, and B NormalImages Based on Plurality of R, G, and B Images (Steps S24 a to 24 c)

The processor 11 generates an R reflection image (a first reflectionimage representing the distribution of reflectances on the surface ofthe object OB of determination) and an R normal image (a first normalimage representing a first distribution of normal directions on thesurface of the object OB of determination) that are associated with thepixels of R, in accordance with the plurality of R images IMr(i) (aplurality of first images) (step S24 a).

Similarly, in accordance with the plurality of G images IMg(i) (theplurality of second images), a G reflection image (a second reflectionimage representing the distribution of reflectances on the surface ofthe object OB of determination) and a G normal image (a second normalimage representing a second distribution of normal directions on thesurface of the object OB of determination) that are associated with thepixels of G are generated (step S24 b). In accordance with the pluralityof B images IMb(i) (the plurality of third images), a B reflection image(a third reflection image representing the distribution of reflectanceson the surface of the object OB of determination) and a B normal image(a third normal image representing a third distribution of normaldirections on the surface of the object OB of determination) that areassociated with the pixels of B are generated (step S24 c).

Optionally, the first distribution, the second distribution, and thethird distribution of normal directions are not represented as imagessuch as the R normal image, the G normal image, and the B normal image.The representation of the distribution of normal directions issufficient for the first to third distributions of normal directions.

In typical photometric stereo, a luminance I(i), a reflectance ρ, thelight source vector L(i), and a normal vector n of a given pixel of theimage IM(i) are in the relationship as indicated by the followingequation (1).

I(i)=ρ(L(i)·n)  Equation (1)

The light source vector L(i) is estimated by step S21, and the luminanceI(i) is determined from the image IM(i). In contrast, the reflectance pand the normal vector n are unknowns. Therefore, simultaneous equationsmade up of a plurality of equations of different light source vectorsL(i) are used to calculate the reflectance p and the normal vector n.

As a result, by determining reflectances p and normal vectors n of allpixels of the image IM(i), it is possible to generate a reflection image(the so-called albedo image) representing the distribution of thereflectances p on the object OB of determination and a shape image (theso-called normal vector image) representing the distribution of thenormal vectors n on the object OB of determination.

In the present embodiment, instead of equation (1), the followingequations (2a) to (2c) are used to be applied to R, G, and B (light ofthe first wavelength band, light of the second wavelength band, andlight of the third wavelength band).

Ir(i)=ρr(L(i)·nr)  Equation (2a)

Ig(i)=ρg(L(i)·ng)  Equation (2b)

Ib(i)=ρb(L(i)·nb)  Equation (2c)

Ir(i), Ig(i), Ib(i): luminances I(i), corresponding to R, G, and B, atone point on the object OB of determination, i.e., luminances,corresponding to R, G, and B, of a pixel of the image IM(i)

-   -   ρg, ρg, and ρb: reflectances ρ, corresponding to R, G, and B, at        one point on the object OB of determination    -   ng, ng, and nb: normal vectors, corresponding to R, G, and B, at        one point on the object OB of determination

It is therefore possible to calculate respective reflectances ρr, ρg,and ρb and respective normal vectors nr, ng, and nb for R, G, and B, inaccordance with the light source vector L(i) (the orientation of thecalculated light source LS(i)). In accordance with the reflectances ρr,ρg, and ρb, it is possible to generate an R reflection imagecorresponding to the pixels of R, a G reflection image corresponding tothe pixels of G, and a B reflection image corresponding to the pixels ofB (eventually in the form of a color reflection image, as will bedescribed later). In accordance with the normal vectors nr, ng, and nb,it is possible to generate an R normal image corresponding to the pixelsof R, a G normal image corresponding to the pixels of G, and a B normalimage corresponding to the pixels of B (eventually in the form of anormal image, as will be described later).

(3) Generation of normal image (shape image) by averaging R, G, and Bnormal images (step S25)

The processor 11 generates a normal image (shape image) by averaging thedistribution (first distribution) of normals in the R normal image(first normal image), the distribution (second distribution) of normalsin the G normal image (second normal image), and the distribution (thirddistribution) of normals in the B normal image (third normal image)(step S25). Thus, it is possible to generate the normal image (shapeimage) by averaging the normal directions in the first distribution, thesecond distribution, and the third distribution.

A normal on the object OB of determination is a quantity correspondingto the shape thereof, and is considered to be basically independent ofthe wavelength of light. Therefore, it is reasonable to organize thenormal vectors nr, ng, and nb for R, G, and B into a single normalvector n. Specifically, the average value of the normal vectors nr, ng,and nb of neighboring places is calculated and defined as the normalvector n. This normal vector n is used to generate a normal imagerepresenting the distribution of normals on the object OB ofdetermination.

The normal image may be generated by removing an outlier from the normalvectors nr, ng, and nb and then averaging the normal vectors nr, ng todetermine the normal vector n. By removing an outlier, the accuracy ofthe normal vector n is improved.

In order for an outlier to be removed, a technique for removing anabnormal light source vector, which will be described later, can beapplied. As an example, a normal vector n that deviates, by apredetermined value (distance) or more, from the acquired average nav ofthe plurality of normal vectors nr, ng, and nb at neighboring places onthe object OB of determination is removed for being abnormal. As anotherexample, DBSCAN may be used to remove an outlier.

Alternatively, the normal image may be generated with use of any of thenormal vectors nr, ng, and nb without averaging the normal vectors nr,ng, and nb.

(4) Generation of Color Reflection Image (Color Image) from R, G, and BReflection Images (Step S26)

The processor 11 composites the R reflection image (first reflectionimage), the G reflection image (second reflection image), and the Breflection image (third reflection image) to generate a color reflectionimage (step S26). Therefore, by using the reflectances ρr, ρg, and ρbfor R, G, and B as the luminances (ρr, ρg, ρb) of a pixel for R, G, andB, it is possible to generate a color reflection image.

As described above, the color reflection image and the normal image aregenerated from the plurality of images IM(i) of the object OB ofdetermination.

In the above description, the reflectances p and the normal vector n arecalculated for each of all pixels of R, G, and B in steps S24 a to 24 c.This processing requires a large amount of computation. With thetechniques described below, it is possible to generate a colorreflection image and a normal image with a reduced amount ofcomputation.

FIG. 4 is a flowchart of another example of the method for generating acolor reflection image (color image) and a normal image (shape image).Since the steps S21 and S22 are the same as those of FIG. 3 , thedescription thereof is omitted.

In this example, the processor 11 makes a plurality of images IM(i)grayscale in monochrome to acquire a plurality of images IM(i) madegrayscale (step S31). The processor 11 then generates a normal image inaccordance with the plurality of images IM(i) having been madegrayscale, in step S32.

In this generation, the normal vector n is calculated using theabove-described equation (1) and an image of the normal vector n iscreated, so that the normal image is generated. Thus, it is notnecessary to calculate the normal vectors nr, ng, and nb for each of R,G, and B. This allows a reduction in the amount of computation. Alongwith the normal vector n, the reflectance p is calculated. Thisreflectance ρ does not need to be used.

As to the color reflection image, an R reflection image, a G reflectionimage, a G reflection image, and a B reflection image are generated fromthe R image, the G image, and the B image extracted (step S23) in thesame manner as in FIG. 3 (step S33 a to S33 c). In this generation, thefollowing equations (3a) to (3c) can be used instead of the equations(2a) to (2c).

Ir(i)=ρr(L(i)·n)  Equation (3a)

Ig(i)=ρg(L(i)·n)  Equation (3b)

Ib(i)=ρb(L(i)·n)  Equation (3c)

It is therefore possible to calculate the reflectances ρr, ρg, and ρbwith use of the normal vector n calculated in step S32. In this case,the normal vector n is known, and only the reflectances ρr, ρg, and ρbare unknowns, accordingly. This allows a significant reduction in theamount of computation.

In this manner, using a plurality of images IM(i) having been madegrayscale allows a reduction in the amount of computation required forgeneration of the color reflection image and the normal image.

Details of the estimation (step S21) of the light source vector isdescribed next. FIG. 5 is a flowchart of an example of the estimationstep of estimating a light source vector. As described above, theestimation step functions as the step of determining the respectiveorientations of the plurality of light sources LS(i).

(1) Selection and Image-Taking of a Light Source (Steps S41 and S42)

The processor 11 selects a light source LS(i) (step S41) and takes animage of a reference object a plurality of times (e.g., 30 times) (stepS42). A plurality of images of the reference object irradiated withlight from a light source LS(i) which are sequentially selected from theplurality of light sources LS(1) to LS(n) are acquired. Prior to thisimage-taking, the reference object is placed in the darkroom BX.

The reference object is, for example, a white plate (e.g., paper), andan object having the normal direction (normal vector n) and thereflectance ρ that are known is typically used. As will be describedlater, in order for a single light source vector L(i) to be determined,image-taking is carried out three times for different orientations(normal vectors n) of the reference object as a set, for example. Thus,“30 times” here makes it possible to calculate a light source vectorL(i) at least 10 times.

(2) Calculation of Plurality of Light Source Vectors (Step S43)

The processor 11 calculates a plurality of light source vectors L(i)corresponding to a light source LS(i) from the plurality of imageshaving been taken (step S43). A plurality of values for the orientation(light source vector L(i)) of the light source LS(i) are determined.

In a case where the normal vector n and the reflectance ρ of thereference object are known, the light source vector L(i) can be easilycalculated using the above-described equation (1). Since a light sourcevector L(i) has three variables (x, y, z), a light source vector L(i) istypically calculated from three different images for three differentorientations (normal direction) of the reference object. As a result,for example, 10 light source vectors L(i) are acquired from 30 images ofthe light sources LS(i).

(3) Removal of Abnormal Light Source Vector and Averaging of LightSource Vectors (Steps S44 and S45)

The processor 11 removes an abnormal light source vector L(i) from theacquired light source vectors L(i) (step S44) and averages the remaininglight source vectors (step S45). In accordance with the light sourcevectors remaining after the removal of an abnormal light source vector,an estimate Lp(i) (the orientation of the light source LS(i)) of thelight source vector L(i) is acquired. A value deviating by apredetermined value or more from the average of the plurality of values(light source vectors L(i)) is removed from the plurality of values(light source vectors L(i)), and in accordance with the plurality ofvalues (a plurality of light source vectors L(i)) excluding the valuedeviating by the predetermined value or more, the orientation of thelight source LS(i) (an estimate Lp(i) of the light source vector L(i))is determined.

The above process is repeated until all light sources LS(i) are selectedand the light source vectors L(i) thereof are estimated (step S46).

FIG. 6 is a view of an example of the approach to removing an abnormallight source vector. The light source vector L(i) is represented as apoint (x, y, z) on the xyz coordinate system. The object OB ofdetermination is disposed at the origin O.

In this example, the light source vectors L(i) are each classified as alight source vector L1(i) falling within a normal range R or as an(abnormal) light source vector L0(i) falling outside the normal range R.However, all of the light source vectors L(i) are of the same lightsource LS(i) and should be identical to each other. However, due tomeasurement or calculation errors, a calculated light source vector L(i)can deviate significantly from the intrinsic value. Thus, by removingthe abnormal light source vector L0(i) in the step of estimating thelight source vector L(i), it is possible to improve the accuracy ofestimation of the light source vector L(i).

In this example, the light source vector L0(i) that deviates by apredetermined value (distance) D or more from the average Lav(i) of theplurality of light source vectors L(i) acquired is removed as anoutlier, and the estimate Lp(i) of the light source vector is calculatedby averaging the remaining light source vectors L1(i).

The distance D may be specified in the form of a number, such as “0.05”.Alternatively, the distance D may be specified to be, for example, 1σ,on the basis of the standard deviation σ of the plurality of lightsource vectors L(i) acquired.

In order for the outlier to be removed, another technique which is, forexample, density-based spatial clustering of applications with noise(DBSCAN) may be used. In DBSCAN, data points are clustered on the basisof the density of coordinates, and a point in the low-density region isremoved as an outlier (noise).

The determination step will be described below, after the description ofa model M1. FIG. 7 is a schematic view of the model M1. The model M1 hasan input layer 21, an intermediate layer 22, and an output layer 23. Theintermediate layer 22 has a plurality of intermediate layers 22(1) to22(n). These intermediate layers 22(i) are, for example, convolutionlayers and pooling layers.

In accordance with at least one selected from the group consisting ofthe color image of the object OB of determination and the shape image ofthe object OB of determination (hereinafter collectively referred to asan image), the model M1 carries out, for example, output of the degreeof similarity to a conforming item, anomaly detection (herein, output ofa score or a heat map), and detection of a defect based on imagerecognition (recognition of an image corresponding to a defect). Forexample, a score or a heat map are outputted in accordance with at leastone feature value outputted from the intermediate layer 22. Although theresult of recognition and detection is outputted from the output layer23, and the feature value is outputted eventually in the form of a scoreor a heat map from the intermediate layer 22 for ease of understanding,only one of these outputs may be carried out, and the output from theoutput layer 23 may be carried out in accordance with the at least onefeature value outputted from the intermediate layer 22.

The model M1 is a trained model of a neural network, and is generated byinput of a plurality of images. By using, as this model M1, thefollowing various models (1) to (4), a conformity-nonconformitydetermination can be made.

(1) Model that Outputs Degree of Similarity to Conforming Item

As the model M1, a neural network model to which a conforming itemitself is mapped can be used to acquire the degree of similarity betweenthe conforming items and the object OB of determination. Examples of themodel M1 can include a common convolutional neural network (CNN) model.

In this case, in the preliminary preparation step (step S11), the modelM1 to which the image of a conforming item of the object OB ofdetermination is mapped is formed by deep learning with use of the imageof the conforming item. This allows calculation of the degree ofsimilarity of the object OB of determination based on the distancebetween the image of the conforming item mapped to the model M1 and aninputted image of the object OB of determination.

When an image of the object OB of determination is inputted to thetrained model M1, the degree of similarity between the conforming itemand the object OB of determination is outputted from the model M1. Forexample, the processor 11 determines that the object OB of determinationis a conforming item when the degree of similarity is greater than apredetermined threshold, and determines that the object OB ofdetermination is a defective item when the degree of similarity is equalto or smaller than the predetermined threshold.

(2) Model that Outputs Result of Anomaly Detection (Score, Heat Map) inAccordance with Features of Object OB of Determination

With use, as the model M1, of a neural network model to which featuresof the object OB of determination that is a conforming item are mapped,it is possible to acquire a result (score or heat map) of anomalydetection based on a feature value (in the intermediate layer 22) of theobject OB of determination. This model M1 is based on the trend ofoutputting, from the intermediate layer 22, feature values approximateto each other for images approximate to each other. Examples of themodel M1 can include Mahalanobis AD, SPADE, PaDiM, PatchCore, andFastFlow.

In this case, in the preliminary preparation step (step S11), the modelM1 is formed by mapping features of a conforming item by deep learningin which the image of a conforming item of the object OB ofdetermination is used. The model M1 functions as the feature extractor.This eliminates the need to map the conforming item itself in the deeplearning, and extracting the feature values of the conforming item issufficient. As a result of the deep learning, it becomes possible tomake anomaly detection (calculation of a score, formation of a heat map)in accordance with a distance between a feature of the conforming itemmapped to the model M1 and a feature of the object OB of determinationoutputted from the intermediate layer 22.

Although the anomaly detection may be carried out in accordance with afeature value itself of the object OB of determination outputted fromthe intermediate layer 22, the anomaly detection may be carried out byinputting this feature value to another model M1. Examples of suchanomaly detection can include FastFlow.

When the image of the object OB of determination is inputted to thetrained model M1, at least one selected from the group consisting of ascore and a heat map is outputted in accordance with at least onefeature value of the object OB of determination outputted from theintermediate layer 22. The score is, for example, a score of the entireimage. Examples of such a score include an anomaly score representingthe degree of abnormality in light of the features of a conforming item.The heat map is a map in the object OB of determination are divided intosegments according to the degree of score. Note that a heat map is oftenused in the course of determination in anomaly detection, and is notnecessarily outputted as the final result of the abnormality detection.

The processor 11 determines whether the object OB of determination isconforming or nonconforming, in accordance with a score or a heat mapthat have been acquired. For example, the processor 11 determines thatthe object OB of determination is a conforming item when the score(anomaly score) is smaller than a predetermined threshold, anddetermines that the object OB of determination is a defective item whenthe score is equal to or greater than the predetermined threshold. In acase of using the heat map, for example, the processor 11 determinesthat the object OB of determination is a conforming item when thesegments of the scores each of which is equal to or greater than a givenvalue cover an area (or the number of pixels) which is smaller than thepredetermined threshold, and determines that the object OB ofdetermination is a defective item when the segments of the scores eachof which is equal to or greater than the given value cover an area whichis equal to or greater than the predetermined threshold.

In Mahalanobis AD, feature values from the intermediate layer 22 aretreated as a multivariate normal distribution, a Mahalanobis distance iscalculated for each of the intermediate layers 22(i) and the sum of theMahalanobis distances is calculated, and the sum is outputted as adistance (score).

In SPADE, PaDiM and PatchCore, a heat map is formed in accordance withfeature values outputted from the intermediate layer 22. In SPADE,feature values outputted from the intermediate layer 22 are treated on apixel-by-pixel basis, and are subjected to comparative classificationbased on the kNN distance to be represented in the form of a heat map.In PaDiM, feature values outputted from the intermediate layer 22 aretreated on a pixel-by-pixel basis, and are subjected to comparativeclassification based on the average and the covariance to be representedin the form of a heat map. In PatchCore, feature values outputted fromthe intermediate layer 22 are subjected to selection, and arerepresented in the form of a heat map on the basis of the nearest value.

Also in FastFlow, feature values are acquired from the intermediatelayer 22, and a result of anomaly detection (e.g., a score, a heat map)is outputted in accordance with these feature values. More specifically,as the model M1, a CNN-based model or a Transformer-based model is usedto create a heat map. Mahalanobis AD, SPADE, PaDiM, PatchCore do notrequire transfer learning while FastFlow requires transfer learning.

(3) Model that Outputs Result of Defect Detection in Accordance withImage Recognition

As the model M1, a model for image recognition that detects a defect ofthe object OB of determination can be used. Examples of such a model M1can include models of you only look once (YOLO), semantic segmentation,and instance segmentation.

In this case, the model M1 which is capable of detecting a defect andthe type of the defect is formed in the preliminary preparation step(step S11) by deep learning in which an image of a defect (e.g.,chipping, foreign matter) to be detected in the object OB ofdetermination.

When an image of the object OB of determination is inputted to the modelM1, the type of a defect (e.g., foreign matter, chipping) detected inthe image is outputted. In this output, in some cases, the portion(region) at which the defect is present is also outputted together. Forexample, the processor 11 determines that the object OB of determinationis a conforming item when a defect is not detected, and determines thatthe object OB of determination is a defective item when a defect isdetected. In a case where the (region) at which the defect is present isoutputted, the processor 11 may determine that the object OB ofdetermination is a conforming item when the area of the defect region issmaller than a predetermined threshold, and may determine that theobject OB of determination is a defective item when the area of thedefect region is equal to or greater than the predetermined threshold.

(4) Combined Use of Models

In the above description, the conformity-nonconformity determination ismade with use of any one selected from the group consisting of the modelM1 for outputting the degree of similarity, the model M1 for anomalydetection (a score, a heat map), and the model M1 for image recognition.A plurality of models M1 may be used in combination for theconformity-nonconformity determination. For example, either the degreeof similarity or anomaly detection can be combined with imagerecognition. This makes it possible to improve the accuracy of thedetermination.

The model M1 (e.g., a typical CNN or PatchCore) that outputs the degreeof similarity or a result of anomaly detection is formed, basically bylearning based on an image of a conforming item. It is therefore easy todetermine that the object OB of determination which contains a defecthaving a color different from the color of a conforming item (e.g., ablack screw or a white paper scrap on a gray casting) is a defectiveitem. However, it is not easy to determine that the object OB ofdetermination that contains a defect having a color close to the colorof a conforming item (e.g., a lack of gray color in a part of a graycasting) is a defective item.

Therefore, by combining the model M1 for outputting the degree ofsimilarity or the model M1 for anomaly detection with the model M1 forimage recognition, it is possible to increase the accuracy of thedetermination. For example, it is possible to makeconformity-nonconformity determination by image recognition and thenmake conformity-nonconformity determination, based on feature values, ofonly the objects OB of determination in each of which no defect has beendetected by the image recognition.

In the above-described embodiment, at least one selected from the groupconsisting of the color reflection image and the normal image may bedivided so that conformity-nonconformity determination may be made foreach of the divisions of the object OB of determination. For example, acolor reflection image and a normal image are divided into four, and atotal of eight images are used for the conformity-nonconformitydetermination. The number of divisions is not limited to 4, but may bemore than or less than 4. For example, the processor 11 may divide analbedo image and a normal vector image into six or nine images.

By dividing the image and making conformity-nonconformity determinationfor each of the segments of the object OB of determination, it ispossible to improve the accuracy of determination as to whether theobject OB of determination is conforming or nonconforming. For example,it may be determined that: the object OB of determination is aconforming item when it is determined that all of the image divisions ofthe object OB of determination are conforming items; and the object OBof determination is a defective item when it is determined that any ofthe image divisions of the object OB of determination is a defectiveitem.

In the embodiment described above, at least one selected from the groupconsisting of the color reflection image and the normal image mayundergo mask processing. For example, regions (e.g., background regions)of the color reflection image and the normal image, the regions notbeing required to be subjected to determination, are filled with a maskimage prepared in advance. This allows an improvement in the accuracy ofthe determination. This mask processing may be performed on a dividedcolor reflection image and a divided normal image.

Collecting images for learning is not necessarily easy because manyimages are required. Therefore, the number of images may be increased byprocessing an image of a conforming item having been taken.

The following description will discuss Examples of the presentinvention. In Examples, the object OB of determination is a casting, andthe appearance of the object OB of determination was determined by theappearance determination device 10. A conforming item and a defectiveitem of the casting were taken by the image-taking section MP so that aplurality of images IM were acquired, and an image (a color reflectionimage in Examples) was generated. By the steps illustrated in FIG. 5 ,the light source vector L(i) of the light source LS(i) was estimated.The casting having foreign matter (screw or paper scrap) placed thereonis defined as a defective item.

The generated images were inputted to the model M1 (models ofsemantic/instance segmentation, YOLO) for image recognition and themodel M1 (Mahalanobis AD and PatchCore) for anomaly detection, so thatwhether the appearance is conforming or nonconforming was determined.Conformity-nonconformity determinations of the appearance with use ofsemantic segmentation and with use of instance segmentation were madeand the results thereof were found to be similar to each other. Thus,semantic segmentation and instance segmentation are collectivelyreferred to as “semantic/instance segmentation” herein.

The model M1 for image detection was trained so as to be capable ofdetecting a defect, by preliminary learning based on images of defects(foreign matter such as a screw and chipping) of the casting. The modelM1 for anomaly detection was trained so as to be capable of outputting ascore or a heat map, by preliminary learning based on the image of aconforming item.

FIG. 8 is a view illustrating examples of appearance determinationresults. An image A1, an image A2, and an image A3 are color images ofthe object OB of determination, and represent a conforming item, adefective item containing foreign matter (screws), and a defective itemcontaining chipping, respectively. An image B1, an image B2, and animage B3 represent the results of determination made with use ofsemantic/instance segmentation, respectively regarding the image A1, theimage A2, and the image A3. Similarly, images Cl to C3, images D1 to D3,and images E1 to E3 represent the results of determination maderespectively with use of YOLO, Mahalanobis AD, and PatchCore, regardingthe images A1 to A3.

As seen in the images B1 to B3 and the images Cl to C3, withsemantic/instance segmentation and YOLO, in which image recognition iscarried out, the images A2 containing foreign matter and the images A3containing chipping were recognized as “foreign matter” and “chipping”.Furthermore, with YOLO, the places at which the “foreign matter” and the“chipping” were present were identified by a square frame (boundingbox). This shows that semantic/instance segmentation and YOLO allowdetermination as to whether the casting is conforming or nonconforming.

As seen in the images D1 to D3, with Mahalanobis AD, the discriminationbetween a conforming item and a defective item was successfully madeaccording to whether the score is great or small. Thus, it is possibleto determine that the object of determination is a defective item when,as a result of comparing a score outputted by Mahalanobis AD with athreshold value (e.g., 120), the score is greater than the thresholdvalue.

As seen in the images E1 to E3, with PatchCore, the discriminationbetween a conforming item and a defective item is made with use of aheat map. Regarding the image A2, determination of anomaly wassuccessfully made from the heat map. However, regarding the image A3, inwhich the color of the chipping portion is close to the color of theother portion, anomaly detection was not reached.

As seen in the images A1 to E1, all of the objects OB of determinationthat are conforming items were determined to be conforming items. Ahundred conforming items were subjected to appearance determination, andall of the items were determined to be conforming items.

As described above, with Mahalanobis AD, both the image A2 (of adefective item containing foreign matter) and the image A3 (of adefective item containing chipping) were successfully determined to bedefective items. On the other hand, with Patch Core, although the imageA2 was successfully determined to be defective, the image A3 was notdetermined to be defective. This may be because the color of the“chipping” of the image A3 is close to the color of the casting itself.

However, it can be understood that, in such a case, the accuracy of theconformity-nonconformity determination can be improved by usingPatchCore in combination with semantic/instance segmentation or YOLO forimage recognition. For example, PatchCore is used forconformity-nonconformity determination of the object OB of determinationhaving a defect that has not been detected with use of semantic/instancesegmentation or YOLO.

In the above Example, a color reflection image is used for appearancedetermination. Alternatively, a normal image or both a color reflectionimage and a normal image may be used for the appearance determination.

FIG. 9 is a view illustrating another example of appearancedetermination results, the view illustrating examples of the appearancedetermination results obtained with use of both the color reflectionimage and the normal image. An image Q1, an image Q2, and an image Q3 ofFIG. 9 are color reflection images of the object OB (casting) ofdetermination, and represent a conforming item, a defective itemcontaining foreign matter, and a defective item containing chipping,respectively. The image Q2 represents a state of foreign matter adheringto the casting. The color of the foreign matter is close to the color ofthe surface of the casting. Images R1, R2, and R3 are normal images ofthe object OB (casting) of determination, and correspond respectively tothe images Q1, Q2, and Q3. Images S1, S2, and S3 represent the results(heat maps) of determination made with use of PatchCore, respectivelyregarding the images R1, R2, and R3. The arrow F indicates a defectiveportion (foreign matter, chipping).

As described above, when the difference in color between a defectiveportion (e.g., foreign matter, chipping) and a non-defective portion issmall (close in color), anomaly detection with use of the colorreflection image is difficult. In this example, the defective portions(foreign matter, chipping) are somewhat unclear in the images Q2 and Q3(color reflection images). This makes it difficult to detect thedefects. In contrast, in the images R2 and R3 (normal images), defectiveportions are clarified as a raised portion (adhesion of foreign matter)and a recessed portion (chipping). As a result, in the images S2 and S3,the defective portions were successfully detected through the heat map.

Thus, it is possible to use a normal image to determine the appearanceof an object OB of determination. A normal image, which representsinformation on irregularities present on an object OB of determination,allows detective portions to be detected as irregularities even if thedefective portions and the non-defective portion are close in color.Further, when both a color reflection image and a normal image are used,a defective portion can be detected more reliably.

As described above, in the present embodiment, whether the object OB ofdetermination is conforming or nonconforming is determined in accordancewith at least one selected from the group consisting of the colorreflection image of the object OB of determination and the normal imageof the object OB of determination.

The present invention is not limited to the embodiments, but can bealtered by a skilled person in the art within the scope of the claims.The present invention also encompasses, in its technical scope, anyembodiment derived by combining technical means disclosed in differingembodiments.

1. An appearance determination device comprising at least one processor,the at least one processor carrying out a determination step ofdetermining whether an object of determination is conforming ornonconforming, in accordance with at least one selected from the groupconsisting of a color image and a shape image, the color image beinggenerated by photometric stereo and representing a color optical imageof the object of determination, the shape image being generated by thephotometric stereo and representing a shape of the object ofdetermination.
 2. The appearance determination device according to claim1, wherein the shape image is a normal image representing a distributionof normal directions on a surface of the object of determination, andthe color image is a color reflection image representing a distributionof reflectances on the surface of the object of determination, thedistribution corresponding to a first wavelength band, a secondwavelength band, and a third wavelength band that are different fromeach other.
 3. The appearance determination device according to claim 2,wherein the at least one processor carries out a generation step ofgenerating the color image and the shape image in accordance with aplurality of images of the object of determination irradiated with lightemitted from each of a plurality of light sources of respectiveorientations different from each other, and determines, in thedetermination step, whether the object of determination is conforming ornonconforming, in accordance with at least one selected from the groupconsisting of the color image generated and the shape image generated.4. The appearance determination device according to claim 3, wherein theplurality of images are each taken with use of light containing light ofthe first wavelength band, light of the second wavelength band, andlight of the third wavelength band, and the at least one processorcarries out, in the generation step, the steps of: extracting, from theplurality of images, a plurality of first images corresponding to thefirst wavelength band, a plurality of second images corresponding to thesecond wavelength band, and a plurality of third images corresponding tothe third wavelength band; generating, in accordance with the pluralityof first images, a first reflection image representing a distribution ofreflectances on the surface of the object of determination; generating,in accordance with the plurality of second images, a second reflectionimage representing a distribution of reflectances on the surface of theobject of determination; generating, in accordance with the plurality ofthird images, a third reflection image representing a distribution ofreflectances on the surface of the object of determination; andcompositing the first reflection image, the second reflection image, andthe third reflection image to generate the color image, which is thecolor reflection image.
 5. The appearance determination device accordingto claim 3, wherein the at least one processor carries out, in thegeneration step, the steps of: making each of the plurality of imagesgrayscale; and generating the shape image, which is the normal image, inaccordance with the plurality of images made grayscale.
 6. Theappearance determination device according to claim 3, wherein theplurality of images are each taken with use of light containing light ofthe first wavelength band, light of the second wavelength band, andlight of the third wavelength band, and the at least one processorcarries out, in the generation step, the steps of: extracting, from theplurality of images, a plurality of first images corresponding to thefirst wavelength band, a plurality of second images corresponding to thesecond wavelength band, and a plurality of third images corresponding tothe third wavelength band; determining a first distribution of normaldirections on the surface of the object of determination, in accordancewith the plurality of first images; determining a second distribution ofnormal directions on the surface of the object of determination, inaccordance with the plurality of second images; determining a thirddistribution of normal directions on the surface of the object ofdetermination, in accordance with the plurality of third images; andaveraging the normal directions in the first distribution, the seconddistribution, and the third distribution to generate the shape image,which is the normal image.
 7. The appearance determination deviceaccording to claim 3, wherein the at least one processor carries out thestep of determining respective orientations of the plurality of lightsources, and generates the color image and the shape image in thegeneration step, in accordance with the plurality of images and theorientations determined.
 8. The appearance determination deviceaccording to claim 7, wherein the at least one processor carries out, inthe step of determining the respective orientations of the plurality oflight sources, the steps of: determining a plurality of values of anorientation of a light source of the plurality of light sources, inaccordance with a plurality of images of a reference object irradiatedwith light emitted from the light source; removing a value from theplurality of values, the value deviating by a predetermined value ormore from an average of the plurality of values; and determining theorientation of the light source in accordance with the plurality ofvalues excluding the value deviating by the predetermined value or more.9. The appearance determination device according to claim 3, wherein theobject of determination is a casting.
 10. The appearance determinationdevice according to claim 1, wherein the at least one processor carriesout, in the determination step, the steps of: inputting at least oneselected from the group consisting of the color image and the shapeimage to a neural network model to which features of the object ofdetermination that is a conforming item are mapped, to acquire a scorebased on at least one feature value of an intermediate layer, or a heatmap in which a plurality of scores based on feature values of theintermediate layer are assigned respective segments; and determiningwhether the object of determination is conforming or nonconforming, inaccordance with the score acquired or the heat map acquired.
 11. Theappearance determination device according to claim 10, wherein in thedetermination step, the at least one processor determines that theobject of determination is a conforming item when the score is smallerthan a predetermined threshold, and determines that the object ofdetermination is a defective item when the score is equal to or greaterthan the predetermined threshold, or determines that the object ofdetermination is a conforming item when the segments of the plurality ofscores each of which is equal to or greater than a given value cover anarea which is smaller than a predetermined threshold, and determinesthat the object of determination is a defective item when the segmentsof the scores each of which is equal to or greater than the given valuecover an area which is equal to or greater than the predeterminedthreshold.
 12. The appearance determination device according to claim 1,wherein the at least one processor carries out, in the determinationstep, the steps of: inputting at least one selected from the groupconsisting of the color image of the object of determination and theshape image of the object of determination to an image recognitionneural network model for detecting a defect of the object ofdetermination, to acquire a recognition result; and determining whetherthe object of determination is conforming or nonconforming, inaccordance with the recognition result acquired.
 13. An appearancedetermination method comprising the steps of: inputting a color imagegenerated by photometric stereo and representing a color optical imageof an object of determination and a shape image generated by thephotometric stereo and representing a shape of the object ofdetermination; and determining whether the object of determination isconforming or nonconforming, in accordance with the color image and theshape image.